Method and apparatus for frame coding in vertical raster scan order for HEVC

ABSTRACT

A method and apparatus for frame coding in adaptive raster scan order. The method includes encoding at least one of image or video utilizing input frames and at least one of a data related to the input frame to produce bitstream with raster scan order information and displacement information for producing compressed video bitstream, at decoding time, decoding at least one of the encoded bitstream with raster scan order information and displacement information for producing compressed video bitstream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. provisional patent applicationSer. No. 61/388,478 and 61/474,435, filed Sep. 30, 2006 and Apr. 12,2011, respectively, which are herein incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to a method andapparatus for frame coding in vertical raster scan for HEVC.

2. Description of the Related Art

HEVC is a video coding standard being standardized to improve codingefficiency by 50% over H264/AVC. To achieve this goal, lots of newcoding tools have been proposed for HEVC. However, all the proposedcoding tools assume the horizontal raster scan of micro blocks, whichhas two problems: 1) Horizontal raster scan is not always effective inprediction especially for intra coding and 2) with sliding window memoryfor motion estimation (ME) and motion compensation (MC), large on-chipmemory is required to compensate for large vertical motion. This isespecially a challenging for UHD (ultra high definition) videos, suchas, 4 k×2 k and 8 k×4 k. While UHD video coding is one of theapplication areas of HEVC, the frame coding in horizontal scan order isnot cost effective at all for UHD videos with high vertical motion.

Therefore, there is a need for a method and/or apparatus for framecoding for HEVC.

SUMMARY OF THE INVENTION

Embodiments of the present invention relate to a method and apparatusfor frame coding in adaptive raster scan order. The method includesencoding at least one of image or video utilizing input frames and atleast one of a data related to the input frame to produce bitstream withraster scan order information and displacement information for producingcompressed video bitstream, at decoding time, decoding at least one ofthe encoded bitstream with raster scan order information anddisplacement information for producing compressed video bitstream.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is an embodiment of a frame coding in horizontal raster scanorder;

FIG. 2 is an embodiment of a frame coding in vertical raster scan order;

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order;

FIG. 4 is an embodiment of a horizontal sliding window scheme forvertical raster coding order;

FIG. 5 is an embodiment depicting horizontal bottom-right to left;

FIG. 6 is an embodiment of a vertical right-bottom to top;

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor;

FIG. 8 is an embodiment of a video stabilizer in accordance with theprior art; and

FIG. 9 is an embodiment of an improved video stabilizer.

DETAILED DESCRIPTION

The proposed method and apparatus add a vertical raster scan order tothe HEVC standard and adaptively select the best coding scan orderbetween the horizontal and vertical raster scan orders based on anycriteria, such as, size of horizontal/vertical motion, codingefficiency, on-chip memory saving and etc.

The vertical raster scan is effective for the frames which have lots ofvertical discontinuities. FIG. 1 is an embodiment of a frame coding inhorizontal raster scan order. FIG. 2 is an embodiment of a frame codingin vertical raster scan order. In FIG. 1 and FIG. 2, the frame withvertical discontinuities with horizontal and vertical raster scan ordersare shown, respectively. When we apply the horizontal raster scan order,as shown in FIG. 1, more bits are used to encode micro block mode. Thisis due to incorrect mode prediction caused by discontinuity. However,the number of bits is reduced when we apply the vertical raster scanorder, as shown in FIG. 2. Hence, the coding gain will be moresignificant for intra frames. The result indicates that the verticalraster scan can reduce bitrates up to 4% for some sequences, evenwithout adaptive horizontal/vertical raster scan order decision. Thus,adaptive decision will result in even higher bit-rate saving.

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order. If a video contains large verticalmotion, the vertical sliding window scheme may not cover the largevertical motion. In this scenario, inter mode may not be used becauserequired reference area is not available. This eventually causesencoding efficiency degradation. However, this problem could be solvedby employing a horizontal sliding window scheme with vertical rastercoding order, as illustrated in FIG. 4. FIG. 4 is an embodiment of ahorizontal sliding window scheme for vertical raster coding order.Utilizing adaptively choosing raster scan order improves the codingefficiency of inter frames at the same on-chip memory requirement. Onthe other hand, the required on-chip memory size without coding gainloss is reduced.

For horizontal and vertical search range srX*srY, on-chip memory size(byte) for sliding window for 8-bit luma can be calculated as follows:MemSizeHorOrder=picWidth*(2*srY+N),MemSizeVertOrder=picHeight*(2*srX+N).

MemSizeHorOrder and MemSizeVertOrder are on-chip memory sizes forvertical sliding window (horizontal raster coding order) and horizontalsliding window (vertical raster coding order), respectively. N*N is thelargest coding unit, and picWidth and picHeight are the horizontal andvertical size of the picture. Based on the equations, Table 1 lists thesearch ranges for different available on-chip memory sizes for 4 k×2 k(3840×2160) videos. For a given on-chip memory size, srY is alwayslimited in vertical sliding window, which potentially causes encodingefficiency degradation for large vertical motion videos as illustratedin FIG. 3. By introducing vertical coding order with horizontal slidingwindow, we can remove the limitation on srY and avoid the degradationfor large vertical motion videos.

TABLE 1 Available on-chip memory sizes vs. search ranges 500 KBytes 750Kbytes 1,000 KBytes srX srY srX srY srX srY Vertical un- +/−33 un- +/−65Un- +/−98 sliding limited limited limited window Horizontal +/−83 un-+/−141 Un- +/−199 un- sliding limited limited limited window

The idea of vertical raster scan can be realized with frame rotation.Hence, similar effect may be seen by rotating input frames withhorizontal raster scan. This information can be simply added as SEI(Supplemental Enhancement information) or VUI (Video UsabilityInformation) in bitstreams. Moreover, we can extend this idea to anyarbitrary raster scan order. We can apply one of 8 different raster canorders for each frames; horizontal top-left to right, as shown in FIG.1, horizontal top-right to left, horizontal bottom-left to right,horizontal bottom-right to left, as shown in FIG. 5, vertical left-topto bottom FIG. 2, vertical left-bottom to bottom, vertical right-top tobottom and vertical right-bottom to top FIG. 6. It also can be realizedby modifying input frames, i.e., e.g. rotating and/or flipping using SEIor VUI.

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor. The motion sensor consists one of the following sensorsor a combination of them: accelerometer, gyroscope, magnetosensor etc.The motion sensor provided the camera orientation or displacementinformation. The displacement can be specified in terms of a rotationmatrix, quarternion, euler angle etc.

FIG. 8 and FIG. 9 show a video stabilization where camera displacementinformation can be used. Video stabilization compensates for jitter invideo due to changing camera position in order to provide a steadyvideo. FIG. 8 is an embodiment of a video stabilizer in accordance withthe prior art. In FIG. 8, the video stabilization is carried out indecoder. The camera displacement parameters are deduced from decodedframes and are used to translates/rotate/warp the decoded frame togenerate stabilized video frames. Calculation of camera displacementparameters from decoded frames is very computationally intensive andprone to incorrect estimation.

On the other hand, FIG. 9 is an embodiment of an improved videostabilizer. The camera displacement information is calculated in thetransmitter by using motion sensors. The displacement information isencapsulated in SEI/VUI and transmitted inside compressed bitstreams toreceiver. The decoder at the receiver decodes the compressed bitstreamwith camera displacement SEI/VUI and generates decoded frames anddisplacement information which is then passed on to the videostabilization module which carries out video stabilization.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method of a digital processor for frame codingin adaptive raster scan order, comprising: encoding at least one ofimage or video utilizing input frames and at least one of a data relatedto the input frame and producing bitstream with adaptive scan orderinformation and displacement information for producing compressed videobitstream.
 2. The method of claim 1, wherein at least one of the dataand the displacement information is embedded in at least one of apicture header, a Supplemental Enhancement Information (SEI) and a VideoUsability Information (VUI) message.
 3. An apparatus for frame coding,comprising: an encoder for encoding bitstream containing at least one ofan adaptive scan order and displacement information for producingcompressed video bitstream.
 4. The apparatus of claim 3 furthercomprising a stabilizer for stabilizing the decoded frames withdisplacement information.
 5. A non-transitory computer readable mediumwith computer readable instructions, when executed perform a method forframe coding in arbitrary raster scan order, comprising: encoding atleast one of image or video utilizing input frames and at least one of adata related to the input frame and producing bitstream with adaptivescan order information and displacement information for producingcompressed video bitstream.
 6. The non-transitory computer readablemedium of claim 5, wherein at least one of the data and the displacementinformation is embedded in at least one of a picture header, aSupplemental Enhancement Information (SEI) and a Video UsabilityInformation (VUI) message.